Chinese Spelling Correction (CSC) commonly lacks large-scale high-quality corpora, due to the labor-intensive labeling of spelling errors in real-life human writing or typing scenarios. Two data augmentation methods are widely adopted: (1) Random Replacement with the guidance of confusion sets and (2) OCR/ASRbased Generation that simulates character misusing. However, both methods inevitably introduce noisy data (e.g., false spelling errors), potentially leading to over-correction. By carefully analyzing the two types of corpora, we find that though the latter achieves more robust generalization performance, the former yields better-calibrated CSC models. We then provide a theoretical analysis of this empirical observation, based on which a corpus refining strategy is proposed. Specifically, OCR/ASR-based Figure 1: Calibration curves and performance of BERTbased data samples are fed into a well-calibrated CSC CSC models trained on random replacement and model trained on random replacement-based OCR/ASR-based data. ECE means the metric of Expected corpora and then filtered based on prediction Calibration Error (Guo et al., 2017), and FPR confidence. By learning a simple BERT-based means the sentence-level false positive rate that measures model on the refined OCR/ASR-based corpus, over-corrections. Combing subplots (a), (b), and we set up impressive state-of-the-art performance (c), OCR/ASR-based data produce better performances on three widely-used benchmarks, while on standard metrics (e.g., P, R, and F1), while random significantly alleviating over-correction (e.g., replacement yields better calibration and FPR.